RoundRobinLoadBalancer fix connection leak during graceful closure #2450
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Scottmitch
commented
Dec 1, 2022
| @@ -640,12 +663,12 @@ previous, new ConnState(newList, newState))) { | |||
| LOGGER.trace("Load balancer for {}: added a new connection {} to {} after {} attempt(s).", | |||
| targetResource, connection, this, addAttempt); | |||
| // Instrument the new connection so we prune it on close | |||
| connection.onClosing().beforeFinally(() -> { | |||
| connection.onClose().beforeFinally(() -> { | |||
There was a problem hiding this comment.
while this is preferred to remove connections from the selection process we still need to manage the lifecycle of the connections. we can use connection.onClosing in the future to disable selection eligibility but the connection by adding extra state or another collection but this PR prioritizes correctness and we can followup to optimize.
idelpivnitskiy
approved these changes
Dec 2, 2022
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Outdated
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Outdated
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Outdated
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Outdated
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Outdated
Show resolved
Hide resolved
servicetalk-loadbalancer/src/main/java/io/servicetalk/loadbalancer/RoundRobinLoadBalancer.java
Show resolved
Hide resolved
Motivation: RoundRobinLoadBalancer clears state of all hosts and each host clears all connections when closing. However a common pattern is to initiate gracefulClosure with a timeout, and then force a full close after a timeout interval. In this scenario RRLB has discarded the reference to connections and will not full close the connections. Modifications: - asyncCloseable needs to keep state about what hosts are being closed and each host can't dereference connections until they are fully closed. - switch back to using onClose to cleanup the connection list as otherwise we may leak connections that have started graceful closure and never finished.
Scottmitch
force-pushed
the
lb_closing
branch
from
7be1407
to
ec4245d
Compare
Scottmitch
force-pushed
the
lb_closing
branch
from
ec4245d
to
aa0def6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
RoundRobinLoadBalancer clears state of all hosts and each host clears all connections when closing. However a common pattern is to initiate gracefulClosure with a timeout, and then force a full close after a timeout interval. In this scenario RRLB has discarded the reference to connections and will not full close the connections.
Modifications: